Papers with model component
Model Compression for Domain Adaptation through Causal Effect Estimation (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for compressing language representation models are not interpretable and do not consider the differences in the predictive power of various model components or the generalizability of the compressed models. |
| Approach: | They propose a model compression scheme that estimates the average treatment effect of a single layer on the model's predictions. |
| Outcome: | The proposed model compression scheme outperforms strong baselines on dozens of domain pairs across three text classification and sequence tagging tasks. |
A Batch Normalized Inference Network Keeps the KL Vanishing Away (2020.acl-main)
Copied to clipboard
| Challenge: | Variational Autoencoder (VAE) is widely used to approximate a model’s posterior on latent variables. |
| Approach: | They propose to let the Kullback–Leibler divergence individual follow a distribution across the whole dataset and analyze that it is sufficient to prevent posterior collapse by keeping the expectation of the KL’s distribution positive. |
| Outcome: | The proposed approach can avoid posterior collapse effectively and efficiently without introducing any new model component or modifying the objective. |
Visually Grounded Continual Language Learning with Selective Specialization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on the role of model components in learning specialized or generalizable representations are lacking. |
| Approach: | They propose to analyze selection strategies for visually grounded continual language learning using two diagnostic datasets. |
| Outcome: | The proposed models outperform existing models and provide enough control and flexibility for a thorough model analysis. |
Towards Detecting Contextual Real-Time Toxicity for In-Game Chat (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ToxBuster is a simple and scalable model that reliably detects toxic content in real-time for a line of chat by including chat history and metadata. |
| Approach: | They propose a model that detects toxic content in real-time for a line of chat by including chat history and metadata. |
| Outcome: | The proposed model outperforms conventional toxicity models across popular multiplayer games including Rainbow Six Siege, For Honor, and DOTA 2 and 6% of unreported toxic players can be proactively moderated. |
Identifying and Mitigating Annotation Bias in Natural Language Understanding using Causal Mediation Analysis (2024.findings-acl)
Copied to clipboard
Sitiporn Sae Lim, Can Udomcharoenchaikit, Peerat Limkonchotiwat, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Current NLU models obtain state-of-the-art accuracy on in-distribution benchmarks, but they use annotation bias to make predictions, negatively affecting the models' generalizability. |
| Approach: | They apply causal mediation analysis to gauge how much each component mediates annotation biases and use causal-grounded masking and gradient unlearning to mitigate bias. |
| Outcome: | The proposed methods improve the model's robustness against annotation bias even after employing other training-time debiasing techniques. |
TPA: Next Token Probability Attribution for Detecting Hallucinations in RAG (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches attribute hallucinations to a binary conflict between internal knowledge stored in FFNs and the retrieved context. |
| Approach: | They propose a framework which mathematically attributes each next-token probability to seven distinct sources and aggregates source attributions by POS tags to quantify contribution of each model component to the generation of specific linguistic categories within a response. |
| Outcome: | Extensive experiments show that the proposed framework achieves state-of-the-art performance. |